Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add logging to worker #1948

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Add logging to worker #1948

wants to merge 10 commits into from

Conversation

anth-volk
Copy link
Collaborator

Fixes #1947. This code:

  • Creates a new class, WorkerLogger, with a variety of methods related to logging, including one to push logs to GCP
  • Creates a second class, MemoryMonitor, that opens a process on a new thread to monitor memory usage, logs usage stats, and logs a warning if system memory usage exceeds 75%
  • Adds logging to worker runs, including accessing of worker and job IDs
  • Refactors compute_general_economy to make it easier to embed a logger into its operations

@anth-volk
Copy link
Collaborator Author

I'm going to have to do some refactoring due to changes made for chunking

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add logging to worker
1 participant